Several works [29], [30] have already implemented vision-based deep learning methods to identify human activities and have shown to be capable of learning features from new sensor data and predicting the associated movement. Most of these studies attempted to improve the performance and accuracy of the deep learning model for human presence and detection activity classification rather than using the data to seek solutions to minimise unnecessary energy loads associated with buildings. Furthermore, no work has attempted to predict the associated sensible and latent heat emission from the occupants, which affects the temperature and humidity levels in an internal space. Furthermore, limited studies conducted tests of vision-based deep learning methods in an actual office environment and assessed its performance in energy savings and indoor environment quality. Finally, the heat emission profiles generated can also be used as input for building energy simulation (BES) tools, increasing the reliability of results since unpredictability of occupant behaviour is one of the parameters that create difficulties for BES.

The present work aims to address the research gaps by using a vision-based deep learning method that enables the real-time detection and recognition of multiple occupants' activities within office building spaces. A faster region-based convolutional neural network (Faster R-CNN) was used to enable training of a classification model which was deployed to a camera for detecting occupancy activities. This method can identify multiple occupants within an indoor space and the activities performed by each. Validation of the developed deep learning model is conducted by using a set of testing data, and the accuracy and suitability for live detection were also evaluated. Experiments are carried out within a case study office room to test the proposed approach's capabilities and accuracy. Using BES, the case study building was simulated with both ‘static' and deep learning influenced profiles (DLIP) to assess the potential energy savings that can be achieved.

Figure 1 presents an overview of the research method. It consists of three main sections. Section 1 (highlighted in green) is the formation and application of a deep learning model for occupancy activity detection and recognition. The model based on a convolutional neural network (CNN) was trained, validated and deployed to an AI-powered camera. Section 2 is the formation of the deep learning influenced profiles (DLIP) using the live occupancy detection within the office space. The DLIP can be fed into a building energy management system and controls of the building heating, ventilation and air-conditioning (HVAC) system to make adjustments based on the actual building conditions while minimising unnecessary loads. However, for the initial analysis (yellow boxes), the DLIP profiles were inputted into building energy simulation to identify potential reductions in building energy consumption and changes within the indoor environment (Section 3). Further details of the steps described in Figure 1 are discussed in the next sub-sections.

Figure 1.

Overview of the proposed framework of a vision-based deep learning method to detect and recognise occupancy activities

Deep learning is a class of machine learning technique utilised to implement object classification, speech recognition, vehicle and pedestrian detection with high accuracy. Compared with other shallow learning methods, deep learning techniques can lead to better performance in detecting and recognising objects. Many studies [31], [32] showed that deep learning models with a convolutional neural network (CNN) based architecture could perform computer vision tasks with high accuracy. Convolutional Neural Network (CNN) is a deep learning network class that is extensively used for image-based classification and recognition applications. Compared with other machine learning-based classification techniques, CNN requires input data in the form of videos or images and can directly feed the data in its original form into the framework model. Instead of performing complex pre-processing stages, the data can be used to derive directly and extract the acquired features from the selected parts of an image [33]. Therefore, CNN algorithm is selected in this study.

In general, the CNN architecture consists of a feedforward network with the input data such as an image is processed through the network. The feature of the data from input images is first extracted within the convolutional layers, and then the spatial volume of the input data is reduced in the pooling layer. The fully connected (FC) layer is then used to classify images between different categories by training. A fully connected layer involves weights, biases, and neurons. The output layer then delivers the outcome of the calculations and extractions. For these layers, the configuration is presented in the form of groups, indicated as stacked modules to present the structure of a deep learning model. The rectified linear unit (ReLU) layer consists of advantages due to its simple function and sparse features, which can minimise training duration. Furthermore, the SoftMax layer provides further constraint to aid the training of the model. Both the ReLU and softmax layers are essential to building CNN architectures for various applications. This includes vision-based applications such as object detection [34] and face recognition [35] and also data analysis and other programmatic marketing solutions [36].

As detailed in [37], [38], the convolutional layers are the first layer to exact features from the input data. It plays a central role in the architecture by utilising techniques to convolve the input data (image). This performs the stages of learning the feature representations while extracting without manual work. Neurons located within each of the convolutional layers are arranged into feature maps. This enables convolution to preserve the relationship between pixels by learning image features using small squares of input data through a mathematical operation. It takes the image matrix and a filter or kernel and passes the result to the next layer through convolutional kernels stride over the whole image, pixel by pixel, to create 3-direction volumes (height, width and depth) of the feature maps.

Then, the ReLU layer introduces nonlinearity into the output neuron. It is an activation function defined as a piecewise linear function that is used to enable direct output when the input was positive or otherwise as a zero output when a negative input is received. According to LeCun, [39], ReLU has become a default activation function for many types of neural networks because a model that uses it is easier to train and often achieves better performance. Through this, the volume size will not be affected while the nonlinear properties of the decision function will be enhanced during this process, resulting in an enrichment of the expressions of an image. Subsequently, the pooling layers enables the reduction in the spatial dimensions of the data (width, height) of the feature maps when the images are too large. For this, the most common spatial pooling type of Max Pooling was selected as it outperforms on processing image datasets [40]. It effectively selects the largest element within each receptive field from left to right, so the output's spatial size is reduced.

Since several convolutional and pooling layers are formed in stacks to enable greater amounts of feature extraction, the fully connected (FC) layers follow on from these layers and interpret the feature representations and perform the function of high-level reasoning to flatten the matrix into a vector form. Combining the features together, the FC layers connect every neuron from one layer to every neuron in another layer. This forms the model, and along with the activation function of SoftMax, it enables the classification of the input images, which generates the classified output results of one of the following occupancy activities.

The exceptional image classification performance of CNN [41], along with its flexibility [42] and popularity within the industry [43] influenced the selection of CNN over other neural network techniques when developing the vision-based occupancy detection and recognition solution. Derived from the understanding of the CNN, Figure 2 presents the CNN based deep learning model configured for the training of the model for occupancy activity detection and recognition. Further discussion of model configuration is outlined within the following sub-sections.

Since this approach is designed to be useful for wider applications to solve other problems related to occupant detection within buildings [44], the deep learning model (Figure 2) was developed and tested following the steps given in Figure 3 to provide a vision-based solution. Part 1 consists of the process of data collection and model training. Images of various types of occupancy activities are collected and processed through manual labelling of the images. Through the analysis of various types of deep learning models, the most suitable type of convolutional neural network-based deep learning model was selected. This was configured specifically for this type of detection approach to provide the model outlined in Figure 2. Next, the model was trained and deployed to an AI-based camera to allow the real-time detection and recognition of occupancy activities, as indicated in Part 2 of the workflow.

Figure 2.

Convolutional Neural Network (CNN) based deep learning model configured for the training of the model for occupancy activity detection and recognition

Figure 3.

The workflow of the deep learning method for model development and application

Data Preparation: Datasets and Pre-Processing

As indicated in Figure 3, the initial stage of the development of the deep learning detection model was to collect relevant input data. Data in the form of images were selected to create large training and testing datasets. For the initial study, the selected data were limited to the most common activities performed in office spaces. The number of images within the datasets followed the rule of thumb and suggestion given by Ng [45]. Table 1 presents the number of images used within the initial development and the images categories based on the selected activity responses. Further development of the method will be carried out in future works by building larger datasets with greater responses and predictions.

All images obtained were pre-processed to the desired format before enabling the data to become ready for model training. The images were manually labelled using the software LabelImg [46]. This is an open-source graphical image annotation tool which allows images to be labelled with bounding boxes to specifically identify the regions of best interest. For some cases, multiple numbers of labels were assigned to each image as this was highly dependent on each image. Hence, the number of labels given in Table 1 was greater than the number of images used. Figure 4 shows an example of the images located within the training and testing datasets of various occupancy activities and how the bounding boxes were assigned around the specific region of interest for each image.

Table 1.

The number of images and labels per category

	Number of Images			Number of Labels
Activity	Training	Testing	Total	Training	Testing	Total
None	100	20	120	108	25	133
Napping	100	20	120	100	20	120
Sitting	100	20	120	146	26	172
Standing	100	20	120	131	26	157
Walking	100	20	120	177	35	212
Total	500	100		662	132

Figure 4.

Example images of various occupancy activities used within the image dataset for training and testing, which were obtained from a relevant keyword search in Google Images; the images were prepared via the labelling of the region of interest (ROI) of each image

This section presents the methods required for the application of the deep learning model. It includes the details of the selected case study building and experimental setup, along with the process of live detection and recognition to form the real-time Deep Learning Influenced Profiles (DLIP).

Case Study Building and Experiment Setup

An office space located on the first floor of the Sustainable Research Building at the University Park Campus, University of Nottingham, UK (Figure 5a) was used to perform the initial live occupancy activity detection using the developed deep learning model. This case study building was also used for the initial performance analysis where the office space was modelled using BES tool IESVE [68] to further assess the potential of this framework and the impact towards building energy loads.

Figure 5c presents the floor plan of the 1^st floor of the building, with the desired office space highlighted. The selected office space consists of a floor area of 39 m² with internal dimensions of 9.24 m × 4.23 m and a floor to ceiling height of 2.5 m. Figure 5b presents the experimental setup with the ‘detection camera' located on one side of the room to enable the detection of occupancy situated on the opposite side. The camera used to generate results for this present study was a 1080p camera with a wide 90 degree field of view. This was connected to a laptop which was operated using the trained deep learning model.

Figure 5.

Sustainable Research Building at University Park Campus, University of Nottingham, UK: photo (a); experimental set up (b); 1^st floor plan (c).

The building operates between the hours of 08:00 to 18:00. This formed the selected hours to perform the experimental occupancy activity detection using the deep learning model. The building is equipped with natural ventilation (manually operated), along with a simple air-conditioning system to provide an internal set point temperature maintained at 21 °C. The Nottingham, UK weather data was inputted into the building energy simulation model. Based on CIBSE Guide A [69], standard occupancy profiles with a sensible and latent heat gain of 70 W/person and 45 W/person was assigned. For the air exchanges, the infiltration rate value was set to 0.1 air changes per hour.

Live Detection and Deep Learning Influenced Profile (DLIP) Formation

Using the developed deep learning model, a typical cold period was selected to perform the live occupancy activity detection and recognition to assess the capabilities of the method. A range of activities was performed by the occupants. This includes the selected desired detection response types of walking, standing, sitting, and none for when no occupants are present. During the real-time detection, the output data for each of the detected occupants were used to form the occupancy heat emission profiles (DLIP). The profile consists of values corresponding to each detected activity and coupled with the heat emission data-based value for an average adult performing the different activities within an office space given in Table 2.

Table 2.

Selected heat emission rates of occupant performing activities within an office [69]

	Rate of Heat Emission
Activity	Total (W)	Sensible (W)	Latent (W)
None	0	0	0
Napping	105	70	35
Sitting	115	75	40
Standing	130	75	55
Walking	145	75	70

Figure 6 shows an example of the process of DLIP formation for the live detection of occupancy activities within the select office space. It presents several snapshots of the recorded frame indicating the detected occupancy activity condition and the percentage of prediction accuracy. A DLIP was formed for each of the detections. This suggests a total of four DLIP would be created for this individual experiment conducted.

As indicated in Figure 5b, the selected office space was designed to accommodate eleven occupants as eleven office workstations were present. However, for the selected experimental test day, only three occupants were present for the majority of the time. This was achieved based on the number of DLIP generated. Effectively, this method not only recognises the activities performed by occupants in forming the desired DLIP but can also obtain data on the number of occupants present in the desired detection space. This could be useful for other types of applications. Further discussion of the detection and recognition of each detection A, B, C and D, along with the detection of each specific activity, is analysed within the corresponding results section.

Figure 6.

Process of forming the deep learning influenced profile from the application of the deep learning approach for occupancy activity detection and recognition

The initial deep learning model was trained using the graphics processing unit (GPU) NVIDIA GeForce GTX 1080. The training approximately took 6 hours 45 minutes for the total losses to reach the level indicated in Figure 7. These training results were obtained using TensorBoard during the training process.

Figure 7.

Deep learning model training results using the Faster-R-CNN with InceptionV2 model over the 6 hours 45 minutes training duration: total loss against the number of training steps (a); total classification loss against the number of steps (b)

Using the Faster-R-CNN with InceptionV2 as the training model, the results provided training for 102,194 steps from a loss of 3.44 to a minimum of 0.01007. Observations made for this proposed approach can be used to compare the performance of different modifications applied in future works. This includes the input of more training and test data and to variations of the type of models for training. Greater amounts of images will be implemented for testing purposes as the framework is developed further.

Based on the images assigned to the test dataset (Table 1), Figure 8 presents an example of the confusion matrix. It shows that majority of the images were correctly classified, showing the suitability of the model for occupancy activity classification. Furthermore, Table 4 presents the model performance based on evaluation in terms of the different evaluation metrics. Overall, it suggests that the classification for ‘none' (when the occupant is absent) achieved the highest performance and ‘standing' achieved the lowest. This perhaps is due to the difficulty in recognising the occupancy body form and shape, as it may be confused with the activities of both standing and walking. Nonetheless, an average accuracy of 97.09% was achieved and an F₁ Score of 0.9270.

Since this model performance evaluation is based on using still test images assigned in the given testing dataset, therefore, the following experimental detection and recognition results can provide more valuable analysis as occupants progressively move, so the detection evaluation is based on a more realistic scenario, including the background conditions, environment setting and realistic occupants behaviour and actions.

Figure 8.

Example of the confusion matrix for occupancy activity classification model

Figure 9 presents example snapshots at various times of the day of the experimental test of the detection and recognition of occupants within the selected office space. Based on the set up indicated in Figure 5b, it shows the ability of the proposed approach to detect and recognise occupants. Up to four output detection bounding boxes were present during this experimental detection, and the accuracy for each detection was also presented above the output bounding boxes. As given by the snapshots in Figure 9, these bounding boxes' size and shape varied between each detection interval. It depends on the size of the detected space, the distance of the camera with the detected person, and it is also dependent on the occupant's activity. In practice, these images will not be saved within the system but real-time data (for example, 1 minute intervals) of occupancy number and activities (heat gains) in the form of numerical and text-based is outputted by the system.

Figure 9.

Example snapshots at various times of the day of the experimental test of the detection and recognition of occupants within an office space using the deep learning occupancy activity detection approach

Figure 10 presents the overall detection performance of the proposed approach during the experimental test. The results showed that the approach provided correct detections 97.32% of the time, 1.98% of the time to achieve incorrect detections and subsequently, 0.70% of the time with no detections. It should be noted that the occupants were asked to carry out their typical office tasks. Overall, this indicates that the selected model provides accurate detections within the desired office space.

Figure 10.

Overall detection performance during the experimental test, identifying the percentage of time achieving correct, incorrect and no detections

Figure 11 shows the results of the detection performances for a) each of the bounding boxes within the camera detection frame and b) for each of the selected response outcome of detected activities. Figure 11a suggests an average detection accuracy of 92.20% for all activities. The highest detection accuracy (98.88%) was achieved for Detection D, and the lowest was observed for Detection A with an accuracy of 87.29%. To provide a detailed analysis of the detection performance, the detections frames from the live detection were identified as Detection A, B, C and D. The results also indicate the ability to identify specific activities which were performed by each occupant during the detection period. However, detection performance cannot be solely be based on the comparison between the results for Detections A – D as not all activities were performed by the detected occupants. Further tests are necessary to fully assess its performance.

Figure 11b presents the detection performance based on the selected activities. Individual detection accuracies for each activity includes walking with 95.83%, standing 87.02%, sitting 97.22% and none (when no occupant is present) achieved an accuracy of 88.13%. This shows the capabilities of the deep learning model to recognise the differences between the corresponding human poses for each specific activity. There is some similarity between the action of standing and walking than there is for sitting. Therefore, this suggests the reason to achieve higher accuracy for sitting as compared to standing and walking.

This section highlights the importance of achieving high accuracy for all activity detections to enable an effective detection approach for building HVAC system controls. Since the following accuracy achieved were only based on small sample size, further model training and testing should be performed to achieve higher detection accuracy for the given occupancy activities to enable further applications of multiple occupancy detection and recognition of a greater number of occupants within different types of office space environments.

Figure 11.

Detection performance based on: each of the bounding boxes within the camera detection frame of Detection A, B, C and D (a); each of the selected response outcomes of detected activities; walking, standing, sitting and none (b)

Figure 12a presents the number of detected occupants in the office space within the office space during the test. Figure 12b shows the number of detected and recognised occupants' activities during the test. This provides a better understanding of the occupancy patterns compared to the data shown in Figure 12a, which highlights the potential of the proposed approach.

Figure 12.

The number of detected occupants in the select office space (a); the number of detected occupants performing each activity during the one-day detection period using the deep learning occupancy detection model (b)

Following the approach detailed in Figure 1, the data obtained from the live detection and recognition of the occupants were used to generate the DLIP. Figure 13 presents the formed DLIP from the experimental activity detection test results. The formation of the profile corresponds to the process indicated in Figure 6, with the activities of Detections A – D. The initial results showed that the DLIP could enable the detection of various activities and provide the identification of times when there are an increase and decrease of activities performed resulting in variation of occupancy heat gains. The DLIP were plotted against the Actual Observation Profile. This defines the ‘actual' occupancy activities performed, which assess the accuracy of the DLIPs. From the comparison of the DLIP and the Actual Observation Profile, an average error of 0.04% was achieved. This indicates the DLIPs would still alternate between the different activities due to the occurrence of prediction error, which suggests the opportunities for further improvements to enhance the accuracy, reliability and stability of the detection model.

Figure 13.

Generated Deep Learning Influenced Profile (DLIP) based on the occupancy activity detection results with the corresponding actual observation for the selected one-day detection

Figure 14 presents two static occupancy profiles typically used in HVAC system operations and in building energy simulations to assume the occupancy patterns in building spaces. Both occupancy profiles were formed assuming that there was constant occupancy in the building spaces and fixed values for occupant internal heat gains. Typical Office 1 represents the average heat gain by a sitting person (115 W). Typical Office Profile 2 represents the average heat gain by a walking person (145 W). During the detection period, there was a 37.38% and 50.25% difference between the Typical Office Profiles 1 and 2 and the Actual Profile. Hence, a large discrepancy between the true occupancy activities performed within the building spaces and the scheduled occupancy profiles can be expected.

Figure 14.

Two static occupancy profiles; Typical Office 1 (sitting) and Typical Office 2 (walking)

The following section provides an analysis of the impact of the proposed deep learning activity detection approach on building energy consumption during a typical winter working day. The generated DLIPs are compared with the static scheduled profiles in Figure 14.

Figure 15 presents the building energy simulation (BES) results of the occupancy sensible and latent gains. Typical office 1 and 2 results followed the assigned static scheduled occupancy profiles (Figure 14). Based on the simulated conditions, it can be observed that the typical office profiles over predicted the occupancy heat gains within the room.

The DLIP results provided a better estimation of the occupancy internal heat gains. The occupancy heat gains were high from 09:00 – 10:00 when there was an increase in activity movement in the space. Lower occupancy heat gains were observed between 13:15 – 13:30 as most of the occupants had left the office space during this time. This shows the potential of the deep learning method in providing a more accurate estimation of the internal heat gains. Additionally, Figure 15b shows the predicted latent heat gains. The accurate prediction of the latent heat gains is important for the estimation of the required dehumidification load and can further reduce unnecessary energy usage. This is important for buildings located in tropical or humid climates as it can lead to heavy usage of air-conditioning systems. The method should be further evaluated by incorporating it into buildings with different climates.

Figure 15.

Occupancy heat gains within the office space during the detection period of 09:00 – 18:00: sensible heat gains (a); latent heat gains (b)

Figure 16 presents a summary of the total sensible and latent occupancy heat gains. Based on the simulated conditions, the occupancy heat gains predicted by using the Typical Office 1 and 2 profiles suggests an overestimation by 22.9% and 54.9% as compared with the Actual Observations. This is equivalent to 83.2 kWh and 199.8 kWh. In comparison, there was a 1.13% (4.1 kWh) difference between the DLIP method and Actual Observations.

Figure 16.

Comparison of the total occupancy heat gains achieved using the deep learning approach in comparison with the different typical occupancy schedules

Figure 17 shows the heating demand of the office space during a typical cold period in the UK, comparing the simulation results of the BES model with different occupancy profiles. Figure 17a presents the heating load across time, and Figure 17b compares the total heating loads for the selected day. The predicted heating load for the model with the DLIP profile was 375.5 kW and was very similar as compared to the Actual Observation profile. While the model with Typical Office 1 and 2 profiles had a heating load of 372.0 kW and 371.8 kW. As expected, the DLIP and actual heat gains in the space were lower than static profiles, which assumed constant activities in the space, and hence the heating requirement will be higher in order to provide comfortable indoor conditions.

Figure 17.

Heating load across time (a); total heating load for a selected typical cold period based on the assignment of the different forms of occupancy profiles – static profiles of Typical Office 1 and 2, ‘true' Actual Observation and the use of the deep learning activity detection approach (b)